Re-enable dollar ($) line anchor in regular expressions in find mode #5289
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Navin Kumar <navink@nvidia.com>
Signed-off-by: Navin Kumar <navink@nvidia.com>
…o enable_dollar_anchor_regex
…able_dollar_anchor_regex
Signed-off-by: Navin Kumar <navink@nvidia.com>
Signed-off-by: Navin Kumar <navink@nvidia.com>
Signed-off-by: Navin Kumar <navink@nvidia.com>
…r line termination characters Signed-off-by: Navin Kumar <navink@nvidia.com>
…able_dollar_anchor_regex
… support to transpile. Also, add more comments Signed-off-by: Navin Kumar <navink@nvidia.com>
Signed-off-by: Navin Kumar <navink@nvidia.com>
|
This is looking good. Should we also include form-feed |
…able_dollar_anchor_regex
…inator to test whitespace around a line anchor Signed-off-by: Navin Kumar <navink@nvidia.com>
|
build |
…able_dollar_anchor_regex
Signed-off-by: Navin Kumar <navink@nvidia.com>
|
build |
andygrove
approved these changes
Apr 25, 2022
3 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #4533.
This re-enables support for the end of line anchor ($) in regular expressions. A couple of caveats:
This only enables support in
RegexFindMode. A separate issue will need to filed for Replace and Split modes (if necessary).Apache Spark actually only uses regular expressions in standard mode (not multiline). Which means that we only support the
$as defined in that mode. This will not match the$in multiline mode.This code handles how the line terminators are managed around the line anchor. The $ has different matching characteristics when used with line terminator characters in the regular expression. Particularly here are a couple of examples:
\r$- this means to only match the CR before the end of the string, so no need to transpile in this case, just match the end of the string using$in cuDF, and the strings are equivalent[any other line terminator character including \n]$- this means match that line terminator character plus optionally any other valid line terminator character before the end of the string$$\n- this means to only match the LF (newline) before the end of the string, and no other line termination sequence. This requires the underlying cuDF to support negative lookahead groups, so this case will fall back to the CPU. (this is because\r\nis a valid line terminator sequence, and this means to forcefully not support that sequence which can only be handled by a negative lookahead group -- See cudf#3100 on lookaheads)[any other line character including \r]$- this means match that line terminator character plus optionally any other valid line terminator character before the end of the string$$$in a row are handled by Java by just reducing to 1$